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ABSTRACT 

As the Illinois public community college system 
approaches its 20th anniversary and in the face of a changing economy 
and shifting public priorities, the establishment of a systematic, 
collegewide program review and evaluation process that can improve 
the colleges' overall effectiveness and efficiency in relation to 
student, community, and state needs has become a necessity. Realizing 
that the primary usefulness of program review is at the individual 
college level and that no two colleges are alike, the Illinois 
Community College Board (ICCB) adopted a revised rule on Program 
Fvaluation and Review, setting minimum standards of complieance and 
the following requirements: (1) the process established by each 
college must be systematic and collegewide; (2) all instructional, 
student, and academic support service programs must be reviewed; (3) 
the program evaluations must focus on the need for the program, the 
quality of the program, and the cost of the program; (4) every 
program must be reviewed at least every 5 years; and (5) each year 
the college must submit a list of programs to be reviewed and a 
summary report to the ICCB. At the state level, both the ICCB and the 
Illinois Board of Higher Education will be involved in the program 
review and evaluation processes. The ICCB will examine each college's 
program review schedule, report of review results, and review 
process. In the context of brief discussions, the report includes 
lists of indicators for assessing program need, costs, and quality, 
along with a sample program review and evaluation report format. 
(EJV) 
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SUGGESTIONS AND GUIDELINES FOR ILLINOIS 
PUBLIC COMMUNITY COLLEGE SYSTEM 
FISCAL YEAR 1986 PROGRAM REVIEW AND EVALUATION 



As the Illinois public community college system approaches its Twentieth 
Anniversary, the years of enrollment growth appear to be coming to a close. 
During the system's infancy, the primary program emphasis was on the 
development of new programs and the expansion of existing programs and 
services to address the needs of a system growing rapidly both in numbers of 
colleges and in numbers of students and potential students* 

In contrast, the system's entry into maturity is signaled by increasing 
stability. Few, if any, new colleges are likely to be founded, and 
system-wide enrollments are leveling off and changing character as declining 
numbers of recent high school graduates are being replaced by greater numbers 
of adults seeking to further their education. At the same time, the nature of 
the job market is changing from a worker-intensive industrial economy to an 
information-processing and service-oriented economy. The context in which 
community colleges must operate, thus, has shifted from spiraling growth in 
all areas to stability in size with a concomitant need to add and modify 
programs to serve new students and the labor market's rapidly changing 
technology . 

In this new context, the college's program decisions and budget decisions 
become inextricably intertwined. The establishment of a systematic, 
college-wide program review and evaluation process can assist the college in 
establishing its programmatic priorities and allocating scarce resources to 
meet them. A systematic program review process can also: 

— Assure students and potential employers of the quality of the degrees 
and certificates awarded by the college; 

~ Assure that the programs provided are relevant to the employment needs 
of the student, the community, and the state; 

— Assure district residents that the programs and services the college 
provides are effective and efficient; 

— Improve the quality of individual programs and services; 

— Improve the college's overall effectiveness and efficiency; and 

~ Provide the information needed for institutional long-range planning 
and individual goal setting. 

Realizing that the primary usefulness of program review is at the college 
level and that no two community colleges in Illinois are alike in size, 
organizational structure, geographic location, or programmatic mix, the 
Illinois Community College Board in March 1983 adopted a revised Rule on the 
Review and Evaluation of Programs. This revised Rule sets a minimum standard 
of compliance for all colleges. The revised Rule essentially requires the 
following five elements: 

1. The process established by each college is to be systematic and 
college-wide . "Systematic" is defined as a definite scheme or method 
of procedure. Thus, a systematic program review and evaluation 
process is one conducted according to a predetermined pattern and 
timetable, with responsibilities for each piece of the whole also 
assigned. 
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"College-wide" means that there is an institutional policy and a 
process that governs program review in all units of the college. Key 
elements of a college-wide process are: a) to determine which programs 
will be evaluated each year and how the evaluations will be conducted, 
b) to analyze the results of the evaluations, c) to develop 
appropriate recommendations, and d) to take action to implement the 
recommendations of the program review. Both "systematic" and 
"college-wide" imply that the process is well documented and 
established as college policy or procedures* 

2. The programs that are to be reviewed include all instructional and 
student and academic support service programs* The word "program 11 was 
chosen specifically because of its broader applicability. 
Instructional programs are commonly called curricula* It may not be 
useful, however, for the college to attempt to review each curriculum 
listed on its Curriculum Master File as a separate entity* In the 
ooccalaureate/transfer area, for example, it may be more advantageous 
to review departments or divisions, while in the occupational area, it 
makes little sense to treat short-term certificate curricula 
separately if they culminate in associate degree curricula. Adult 
education, remedial education, and continuing education (both credit 
and non-credit) may be viewed more appropriately as "functions" if 
each is organizationally dispersed throughout the college. The 
various student services also might be viewed more appropriately as 
"functions" for the same reason. The college's organizational 
structure should help, thus, define "program." 

3. The college needs to include within its review criteria program need, 
program quality, and program cost. In keeping with the notion of 
systematic and college-wide review, the college will need to select 
indicators of these three criteria that appropriately reflect each of 
the college's various programs. 

A. Every program is to be reviewed at least every five years. This time 
period was selected because it coincides with the present Illinois 
State Board of Education's Department of Adult, Vocational and 
Technical Education review requirement for occupational programs and 
the Illinois Board of Higher Education review requirement for public 
universities. Shorter cycles are, of course, possible and may be 
necessary to accommodate external specialized accrediting and funding 
agencies. 

5. Each August 1, the college is to submit to the Illinois Community 
College Board: a) a listing of the programs it intends to review 
during the coming year and b) a report summarizing the findings and 
the actions taken on each of the previous year's program reviews. 

By stating the Rule as a minimum standard, the Illinois Community College 
Board permits each college the flexibility within these parameters to design a 
program review and evaluation process to serve the college's unique needs and 
situation, while at the same time achieving some consistency among colleges. 
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Suqgested Indicator s 

While program review and evaluation should be designed to serve the needs of 
each college, the criteria used in the college's program review and evaluation 
process must include need, cost, and quality. There is usually no single 
standard of measurement (indicator) for these criteiia. The data that are 
assembled and the analyses that are carried out to support decisions at the 
college level should be tailored to meet the college's needs and designed to 
help evaluate the defined objectives of a program. There are, however, 
certain indicators that generally are available and that would be appropriate 
for all colleges. The specific form of each may vary slightly among the 
diverse types of programs. These basic indicators are relevant for program 
review at the state level. It is important to review the criteria indicators 
using trend (historical, current, and projected) data. To review data for 
only one point in time can be very misleading. Likewise, reviewing trends for 
only one indicator at a time can produce invalid conclusions. Thus, for the 
review process to be effective, it should involve the interrelationships of 
various indicators over time. 



Need 

Need for a program can be reviewed from several aspects depending upon the 
type of program being reviewed. Evidence of program need by students is 
common to all programs. For some programs, however, student need for various 
components of a program rather than need for the program itself could be 
reviewed. For example, if student services are being reviewed as a program 
area, there is little doubt of the student need for the program as a whole. 
However, there may be various components of student services for which the 
student need differs dramatically. 

Need for occupational programs should be reviewed for evidence of need by both 
students and potential employers. Once again, need for components of the 
program as well as the program as a whole can be reviewed. It may be 
determined through examination of the indicators that neither student nor 
employer demand indicates the certificate component of a particular program is 
needed but that need for the associate degree component is strong. 

The table on the next page lists possible indicators of need for the program 
being evaluated. As the table suggests, some indicators may be more 
appropriate or more easily available for certain kinds of programs. Much of 
the data are available from each college's computerized data files, college 
reports, and the Illinois Community College Board MIS or other reports. Other 
information is available from other state agencies such as the Illinois 
Department of Employment Security and the Illinois Occupational Information 
Coordinating Committee. It should be noted that the review should not be 
limited necessarily to these indicators. 
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, Type of Program 

Academic 

Bacca- Occupa- Other Student Support 

laureate tional In s true.** Services Services 



*Historic and projected 
full- and part-time annual 
program enrollment 

*Historic and projected 
number of program 
completers 

Historic and projected 
number of credit hours 
produced in program area 

Historic and projected 
employer demand for 
program completers 

Number of students and 
faculty using the service 
per month 

Ratio of monthly usage 
to total enrollment/ 
faculty 

Accessibility/timeliness 
of services X X 

*Historic data are available, in part, from the ICCB MIS 
**Includes remedial, ABE/ASE, General Studies, continuing education, and 
community education. 



Cost 

Cost is a very important criterion to consider in program review. No 
individual program or college can operate without considering the cost of 
various alternatives. Program costs should be reviewed with care. Some 
programs cost more than others. Often there are justifiable reasons for 
higher costs. Sometimes low costs may indicate inadequate equipment and 
resources and, thus, are not desirable. Therefore, although cost indicators 
are very essential in program review, a given dollar amount does not, by 
itself, serve as an indicator. 

In attempting to identify the most basic cost indicators for a program, the 
costs which are unique to the given program are most useful* These types of 
cost tend to show how much extra the college is spending to offer the program. 



Indicators of Program 
Need 



X 
X 
X 



X 
X 
X 
X 



X 
X 
X 
X 

X X 
X X 
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The table below lists possible indicators of costs of the program being 
evaluated and the kinds of programs to which the indicators are applicable* 



Type of Program 



Indicators of Program 
Costs 

^Historic and projected 
annual unit cost per 
credit/contact hour 

Historic annual cost 
per program completer 

Historic number of 
faculty/staff — full- 
and part-time and PTE 

Historic course section 
size (mean and range) 

Historic faculty/staff 
to student ratio (in 
FTE) 

Historic and projected 
equipment costs 

Historic and projected 
faculty/staff costs 



Bacca- 
laureate 



Occupa- 
tional 



Other 

Instruc.** 



Student 
Services 



Academic 

Support 

Services 



^Historic data are available, in part, from the ICCB MIS 
**Includes remedial, ABE/ASE, General Studies, continuing education, and 
community education. 



Quality 

Quality is perhaps the most important criteria in program review; however, il 
is often much more difficult to evaluate than need or cost because the 
indicators of quality are frequently less quantifiable than those of need and 
cost* To evaluate the quality of a particular program, the objectives of that 
program should be well defined and each indicator examined in light of those 
objectives. The table on the next page lists only a few of the suggested 
indicators of quality. As was the case with the possible indicators of need 
and cost, the list is not intended to be all inclusive. Quality indicators 
are commonly classified by type into the three categories of incut, process, 
and outcome. It is essential that a program review include an appropriate 
balance of indicators of quality from each of these categories. 
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Type of Program 



Indicators of Program 
Quality 

INPUT INDICATORS 

Faculty/staff qualifi- 
cations and availability 
of additions/replacements 

Program admissions/place- 
ment criteria 

PROCESS INDICATORS 

Student, peer, and 
supervisor ratings of 
teaching/service provided 

Quantity, age, condition 
and appropriateness of 
equipment and materials 

Quantity, condition, and 
appropriateness of 
facilities 



Bacca- 
laureate 



Occupa- 
tional 



X 
X 



Academic 

Other Student Support 

Instruc.** Services Services 



X 
X 



Student rating of 
program (current and 
follow up) 

Business/industry 
employer rating of 
program 

Results of specialised 
accreditation reviews 

Attrition/retention rates 

Appropriateness of 
course sequencing 

Course articulation 

Historic annual ratio 
of completers to PTE 
enrollment 



X 
X 



X 
X 

X 
X 



X 
X 
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Indicators of Program 
Quality 



Type of Program 



Bacca- Occupa- 
laureate tional 



Other 
Instruct 



Student 
Services 



Academic 

Support 

Services 



OUTCOME INDICATORS 

♦Historic annual number 
of completers employed 
full- and part-time in 

related field X X 

♦Annual salary of 
completers employed full- 
time in related field X X 

♦Historic annual number 
of completers continuing 

their education XXX 

Percentage of completers 
who pass licensing or GEO 

exam, if applicable X X 

♦Historic data are available, in part, from the ICCB MIS 
♦♦Includes remedial, ABE/ASE, General Studies, continuing education, and 
community education. 

Review of Programs at the State Level 

Both the Illinois Community College Board and the Illinois Board of Higher 
Education will be involved in the program review and evaluation process at the 
state level. This section summarizes the process that occurs once the 
college's current program review schedule and the prior year's program review 
results are received at the Illinois Community College Board office on August 1 
of each year. 

Overview of ICCB Review of College's Program Review 

The ICCB staff will review each college's program review process in the 
following three ways: 

1. Review of each college's program review schedule for the current year 
and preceding years; 

2. Review of each college's report of the prior year's program review 
results; and 

3. Review of each college's program review process during the ICCB 
recognition visit (FY 1986-88 recognition cycle). 
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Review of the Program Review Schedule 

The review of each college's program review schedule for the current year will 
include a check of the programs that were reviewed in prior years. The key 
items checked in this review are the following: 

1. Is the college likely to meet the five-year cycle requirement? 

2. Are all types (1.1, 1.2, etc.) and levels of instructional programs, 
as well as student and academic support programs, represented? 

If the college modifies its review schedule during the year ? it should notify 
the ICCB of such modifications. 



Review of the Program Review Annual Report 

A report of each college's prior year's program review results is required by 
the ICCB by August 1, following the end of the fiscal year. A suggested 
format for this report can be found in Appendix A. This report is reviewed by 
the ICCB staff using the following parameters: 

1. A comparison with the prior year's list of programs to be reviewed or 
reported modifications to the schedule will be made to determine 
whether the college reviewed the programs it indicated it would. Any 
differences, such as whether the college added, subtracted, or 
substituted programs from its previous proposed schedule will be noted. 

2. The college's report will be reviewed to determine if the College met 
the parameters of the ICCB Rule. The following questions will be 
asked in this process: 

~ Was the process "ct llege-wide" and "systematic"? That is, was a 
review made beyond an individual unit self-study? 

— Were indicators of program need, cost and quality used? 

— • Were criteria indicators reasonably consistent from program to 
program? 

3. Oecisions made on the program review will be examined to determine: 

— How many programs will be: 

a. continued without modifications; 

b. modified (consolidated, merged, expanded, reduced); 

c. scheduled for further review during the next year; 

d. phased out/withdrawn? 

— Were programs slated for withdrawal, consolidation, or expansion 
so indicated on the RAMP/CC Table 4.1 

A. The quality of the decisions made in the college's review process will 
be reviewed by examining the following criteria: 

~ Were the appropriate criteria indicators used? 

— Did the college attempt to meet the intent of the Rule (even 
though it may not have succeeded)? 

-- - Did the results of the review justify the recommendations/actions? 
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Following this review of the summary report, a letter will be written to each 
college identifying any problems or concerns. The college may be asked to 
supply further information if the college's report does not contain sufficient 
information to evaluate adequately the review based on the above questions. 

The ICCB staff will then prepare a statewide report on the program review 
process that has been completed in the community college system during the 
prior year. This report will be presented to the ICCB and transmitted to the 
Illinois Board of Higher Education. Although the exact nature of each year's 
report may vary, the report will address the following: 

1. An introduction summarizing the ICCB Rule on program review and its 
intent; 

2. A listing of programs reviewed by each college during the year and 
decisions made by the colleges on these programs; and 

3. An analysis of the colleges' program review processes and results with 
possible recommendations for future action to the colleges and/or the 
ICCB (including any Rules changes deemed necessary). 

In addition to this general review of the college's summary reports, the ICCB 
staff, in cooperation with the Board of Higher Education staff, will select 
certain curricular areas for intensive st^ce-level review each year. The 
Beard of Higher Education has a statutory responsibility to "...review 
periodically all existing programs of instruction, research and public service 
at the state universities and colleges and to advise the appropriate board of 
control if the contribution of each program is not educationally and 
economically justified." To discharge this responsibility the IBHE will rely 
on using the ICCB program review mechanism and this intensive state-level 
review. The areas for intensive review will be selected on the basis of 
statewide data on such factors as enrollment trends, employment demand, and 
placement rates as well as on special issues that may be of concern at the 
state level. The colleges' summary reports for the selected areas will be 
analyzed in light of these factors, and specific programs may be compared to 
statewide data for similar program areas. This analysis will rely heavily on 
available ICCB/MIS data about a particular program ana about similar programs 
throughout the state. These data include enrollments, completions, unit cost, 
and follow-up study results. In some cases, the Board of Higher Education 
staff and the Illinois Community College Board staff may request additional 
information from colleges to support their review conclusions. Any requests 
for such information will be transmitted to the colleges by November 15 of 
each year. Resoonses to any staff inquiries concerning selected programs 
should be submitted by January 15. 

Based on this special analysis, the IBHE staff will prepare an annual report 
of staff conclusions and recommendations. A draft of this report will be 
shared with the ICCB staff in March. The ICCB staff will have an opportunity 
to comment on the report. A revised report and recommendations will be 
submitted to the IBHE in May. The report will present the results of the 
cor.minity colleges' program reviews and the staff's conclusions resulting from 
the special analysis. Recommendations will identify those programs (if any) 
that in the judgment of the IBHE staff are not educationally or economically 
justified. The IBHE will notify the district Board through the ICCB of these 
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programs and will recommend that the district Board take appropriate action. 
Although the IBHE recommendations are not mandatory, the district Board must 
report to the ICCB on any subsequent actions taken by the college. 



ICCB Recognition Review 

The ICCB staff will review each college's program evaluation process in 
conjunction with the college's recognition visit. The current ICCB 
recognition visit schedule is based on a three-year cycle with each college 
scheduled for a visit every third year. During the FY 1986-88 visit cycle the 
ICCB staff will review the college's program review and evaluation process to 
determine if it is in compliance with ICCB Rules by focusing on the following 
key items: 

— Does the college have on file a copy of the process documentation and 
individual program reviews? 

— Did the college submit to ICCB a list of programs to be reviewed and a 
report of program reviews by August 1 of each year? 

— Is the program evaluation process "college-wide"? 

— Is the program evaluation process "systematic"? 

— Were indicators of program quality, need, and cost used? Were 
criteria indicators reasonably consistent from program to program? 
Were the indicators appropriate? 

— Were results of program review used in the college's decision-making 
process? 

— How many program changes, additions, or deletions resulted from the 
program review process? Were they appropriate? Were plans for 
improvement carried out? 

The ICCB staff will use the following resources in the recognition process: 
the college's process description and instrumentation, individualized program 
reviews, program review schedule listings and program review reports to ICCB, 
data submission schedules, and MIS and other ICCB reports. 
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APPENDIX A 

SAMPLE PROGRAM REVIEW AND EVALUATION REPORT FORMAT 
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APPENDIX A 

SAMPLE PROGRAM REVIEW AND EVALUATION REPORT FORMAT 



I. Overview of year's review and evaluation process (How the year's review 
fits into the college's five-year review cycle, as well as a brief 
description of the college-wide, systematic process) 

II, Summaries of individual program reviews (One summary for each 
program/unit reviewed): Each summary should contain the following: 

A. Name of program area (with CIP code(s), if appropriate). If the 
program title includes several curricula, list all curricula 
included. 

B. Need information 

1. Indicators used 

2. Results and conclusions 

C. Cost information 

1. Indicators used 

2. Results and conclusions 

0. Quality information 

1. Indicators used 

2. Results and conclusions 

E. Recommendations based on results and conclusions 

F. Actions talon or to be taken with timelines for each action 
III. Summary of college-wide annual review results/actions 

IV. List of programs to be reviewed during the current fiscal year 
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